Search for: All records

A high-quality hydrographic observational database is essential for ocean and climate studies and operational applications. Because there are numerous global and regional ocean databases, duplicate data continues to be an issue in data management, data processing and database merging, posing a challenge on effectively and accurately using oceanographic data to derive robust statistics and reliable data products. This study aims to provide algorithms to identify the duplicates and assign labels to them. We propose first a set of criteria to define the duplicate data; and second, an open-source and semi-automatic system to detect duplicate data and erroneous metadata. This system includes several algorithms for automatic checks using statistical methods (such as Principal Component Analysis and entropy weighting) and an additional expert (manual) check. The robustness of the system is then evaluated with a subset of the World Ocean Database (WOD18) with over 600,000in-situtemperature and salinity profiles. This system is an open-source Python package (named DC_OCEAN) allowing users to effectively use the software. Users can customize their settings. The application result from the WOD18 subset also forms a benchmark dataset, which is available to support future studies on duplicate checks, metadata error identification, and machine learning applications. This duplicate checking system will be incorporated into the International Quality-controlled Ocean Database (IQuOD) data quality control system to guarantee the uniqueness of ocean observation data in this product. 

Millions of in situ ocean temperature profiles have been collected historically using various instrument types with varying sensor accuracy and then assembled into global databases. These are essential to our current understanding of the changing state of the oceans, sea level, Earth’s climate, marine ecosystems and fisheries, and for constraining model projections of future change that underpin mitigation and adaptation solutions. Profiles distributed shortly after collection are also widely used in operational applications such as real-time monitoring and forecasting of the ocean state and weather prediction. Before use in scientific or societal service applications, quality control (QC) procedures need to be applied to flag and ultimately remove erroneous data. Automatic QC (AQC) checks are vital to the timeliness of operational applications and for reducing the volume of dubious data which later require QC processing by a human for delayed mode applications. Despite the large suite of evolving AQC checks developed by institutions worldwide, the most effective set of AQC checks was not known. We have developed a framework to assess the performance of AQC checks, under the auspices of the International Quality Controlled Ocean Database (IQuOD) project. The IQuOD-AQC framework is an open-source collaborative software infrastructure built in Python (available from https://github.com/IQuOD ). Sixty AQC checks have been implemented in this framework. Their performance was benchmarked against three reference datasets which contained a spectrum of instrument types and error modes flagged in their profiles. One of these (a subset of the Quality-controlled Ocean Temperature Archive (QuOTA) dataset that had been manually inspected for quality issues by its creators) was also used to identify optimal sets of AQC checks. Results suggest that the AQC checks are effective for most historical data, but less so in the case of data from Mechanical Bathythermographs (MBTs), and much less effective for Argo data. The optimal AQC sets will be applied to generate quality flags for the next release of the IQuOD dataset. This will further elevate the quality and historical value of millions of temperature profile data which have already been improved by IQuOD intelligent metadata and observational uncertainty information ( https://doi.org/10.7289/v51r6nsf ). 

Abstract Despite technological advances over the last several decades, ship-based hydrography remains the only method for obtaining high-quality, high spatial and vertical resolution measurements of physical, chemical, and biological parameters over the full water column essential for physical, chemical, and biological oceanography and climate science. The Global Ocean Ship-based Hydrographic Investigations Program (GO-SHIP) coordinates a network of globally sustained hydrographic sections. These data provide a unique data set that spans four decades, comprised of more than 40 cross-ocean transects. The section data are, however, difficult to use owing to inhomogeneous format. The purpose of this new temperature, salinity, and dissolved oxygen data product is to combine, reformat and grid these data measured by Conductivity-Temperature-Depth-Oxygen (CTDO) profilers in order to facilitate their use by a wider audience. The product is machine readable and readily accessible by many existing visualisation and analysis software packages. The data processing can be repeated with modifications to suit various applications such as analysis of deep ocean, validation of numerical simulation, and calibration of autonomous platforms. 

Ocean temperature observations are crucial for a host of climate research and forecasting activities, such as climate monitoring, ocean reanalysis and state estimation, seasonal-to-decadal forecasts, and ocean forecasting. For all of these applications, it is crucial to understand the uncertainty attached to each of the observations, accounting for changes in instrument technology and observing practices over time. Here, we describe the rationale behind the uncertainty specification provided for all in situ ocean temperature observations in the International Quality-controlled Ocean Database (IQuOD) v0.1, a value-added data product served alongside the World Ocean Database (WOD). We collected information from manufacturer specifications and other publications, providing the end user with uncertainty estimates based mainly on instrument type, along with extant auxiliary information such as calibration and collection method. The provision of a consistent set of observation uncertainties will provide a more complete understanding of historical ocean observations used to examine the changing environment. Moving forward, IQuOD will continue to work with the ocean observation, data assimilation and ocean climate communities to further refine uncertainty quantification. We encourage submissions of metadata and information about historical practices to the IQuOD project and WOD.